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An Academic Department’s Response to Outcome Assessment 

Abstract 

Institutions of higher education are responding to increasing demands 
for accountability at all levels of government. The demands are also coming 
from regional accrediting associations in the form of calls for institutions 
to document their efforts in the areas of effectiveness and outcomes 
assessment. This paper describes briefly this accreditation requirement. It 
presents one instructional department’s response to outcomes assessment and 
student academic achievement and the direction it has taken for self 
assessment and improvement. Several models of outcomes assessment are 
discussed as are the detractors to self improvement in teaching that are 
presented by the mission of a research institutions. 
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Accreditation Agencies and Outcomes Assessment 

In the very recent past we have heard in an increasingly louder voice 
the call for "accountability" of educational systems at all levels. Demands 
for accountability by higher education have found their way into the 
requirements of all of the regional accrediting bodies . 

The Manual of Criteria for Accreditation of the Commission on Colleges 
for the Southern Association of Colleges and Schools lists over 120 "must" 
statements, prescribing very specifically the requirements that a college or 
university must demonstrate in order to achieve or maintain accreditation. 
Section III of the Criteria speaks totally to institutional effectiveness. 
This section states, "the institution must establish adequate procedures for 
planning and evaluation. The institution must define its expected educational 
results and describe how the achievement of these results will be ascertained" 
(SACS, p. 16) . This section has lent some confusion, incidentally, to the 
issue of accountability and how we measure what we do. The confusion stems 
from the word effectiveness. That is, are we doing what we say we are 
supposed to do, i.e., graduating students, doing research, and providing 
public service, which are measures of effectiveness fairly easy to quantify, 
or are we imparting knowledge, which is sometimes harder to determine. 
Continuing with the Southern Association, this confusion in purpose is 
somewhat cleared up. Later, the criteria specify, "... institutions should 
ascertain periodically the change in the academic achievement of their 
students" (SACS p. 16) . 

The Middle States Association Commission on Higher Education has 
incorporated the outcomes assessment issue into its self-study program for 
accreditation . 

The Western Association of Senior College has made institutional 
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effectiveness a standard for accreditation. 

In October, 1989, our own North Central Association of College and 
Schools began to document its initiative on outcomes assessment, or as it is 
referred in NCA documents, student academic achievement. The issue of student 
academic achievement is addressed in the third and fourth criteria: "The 
institution is accomplishing its educational and other purposes" and "The 
institution can continue to accomplish its purposes and strengthen its 
educational effectiveness" (NCA, 1994, p. 34-5) . 

Finally, the federal government has made its own foray into student 
academic achievement, or outcomes assessment; at least, accrediting bodies are 
sensing this intrusion. Goal five of the National Education Goals calls for 
"all citizens to be literate and able to exercise the rights and 
responsibilities of citizenship" (Rodriguez and Nettles, 1993) . This goal, to 
some, poses an interface between institutional effectiveness, outcomes 
assessment, and national priorities. One possible outcome of this interface 
can be increased mandates for accountability by state agencies . One example 
of such a trickle down effect can be found in Florida. In 1992 the Florida 
state legislature directed that an accountability study be undertaken to 
assess the status of outcomes assessment in Florida institutions of higher 
education. This assessment resulted revised state statutes that specify 
effectiveness and accountability (Florida State Pos t seconda ry Education 
Planning Commission) . 

Prior to 1985-89, regional accreditation associations judged 
institutions for accreditation on the basis of input. That is, if colleges 
and universities had adequate classroom space, suitable facilities and 
laboratories, the proper number of books in the library, faculty with the 
right credentials, and a financial base sound enough to stay in business, they 
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were judged to be accomplishing their missions. The push for accountability 
at all levels moved accreditation into the institutional effectiveness model. 
Whether the associations called it effectiveness or outcomes assessment, the 
issue was student academic achievement. Astin (1992) hails this departure from 
the resource model of excellence toward a student outcomes assessment as a 
positive move. He states, "it directs attention away from mere resource 
acquisition and more toward the talent development process" (p. 305) . 

To a casual spectator outside the education arena, this move toward 
outcomes assessment would appear to have created a whole new industry. We have 
new books on the topic, consultants willing to help, conference themes and 
entire conferences on the subject, and last but not least, for those of us 
struggling with it, new opportunities to write on the subject and add 
publications to our credit. This last is somewhat ironic in that while 
outcomes assessment gives us at universities an imperative to focus on 
teaching, we see it more as an enhancement of opportunities to do research. 
Therein lies a dilemma in the making. On the one hand we have an 
accountability systems that demands that much more attention be placed on 
teaching and on the assessment of the effectiveness of that teaching. On the 
other, by its time and energy requirements, it creates conditions fostering 
its mediocrity by competing with research requirements in a reward system that 
lends more weight to research than to teaching. 

Outcomes Assessment Models and Quality Measurement 

As outcomes assessment techniques are increasingly applied in higher 
education environments, each institution must address several basic issues: 
what model of outcomes based assessment to apply; how quality will be defined 
and measured; and how the results of the assessment process will be used at 
various organizational levels within the university. As in any large complex 
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university, the outcomes assessment process and guidelines had been filtered 
through several organizational levels prior to being submitted to our faculty 
for consideration and action. In developing our department’s outcomes 
assessment process, we considered the various perspectives that could be 
utilized in the development of an institutional outcomes assessment plan, 
attempted to place our university’s choices indicated by the guidelines we 
received within a frame of reference familiar to us, and reflected on how the 
university guidelines could be most meaningfully applied in our own department 
setting and within our program mission, goals, and objectives. More simply 
stated, we tried to figure out what we were being asked to do and how to 
relate these tasks to familiar concepts. 

In 1987, Ewell discussed four main functions or purposes for outcomes 
based education: 

mission review and analysis, including specification of the kinds of 
instructional outcomes intended; 

evaluation of the effectiveness of general education programs; 

evaluation of the outcomes of individual degree programs; 

evaluation of student satisfaction and individual goal attainment; and 

utilization of assessment results in institutional decision making 
processes. (p.5) 

Within this broad spectrum, however, various models may be applied and quality 
measures may also be variously defined. While models for outcomes assessment 
may be viewed as discrete conceptual frames, in reality it is common for 
multiple models to be simultaneously utilized by a single institution. The 
choice of model and the manner of its application reflects the management 
preferences or leadership styles of central administrators at the institution, 
the history and tradition of student and faculty involvement in programmatic 
decision-making, the audience receiving the information, its intended use, and 
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the analytical sophistication of staff supporting the outcomes assessment 
effort . 

The outcomes assessment models that had been most commonly applied in 
our department were the connoisseurship and the responsive models. The 
responsive model is most commonly used in program effectiveness reviews 
(Conrad and Wilson, 1986, p.2) . This model focuses attention on the issues 
and concerns of those involved in the program and had been the model most 
commonly applied in our department. Periodically, we applied some form of 
survey technique to consult with recent graduates of our program and solicited 
their opinions about the appropriateness of our program and the usefulness of 
the program in their applied professional settings. We also consulted with 
hiring authorities regarding their perceptions of the success of program 
graduates. Specifically, administrators in the public schools, community 
colleges, and universities who had worked with our graduates were asked to 
rate our program’s effectiveness. 

The connoisseurship model is most commonly associated with accreditation 
reviews and incorporates concepts of expert power and peer review applied from 
an external source of accountability. While periodic reviews by accrediting 
bodies are common, they are not particularly welcome occasions because of the 
negative consequences that can result from loss of accreditation and national 
recognition. Despite the emphasis on the peer review concept and its implied 
tie to collegiality, it is often difficult to create an atmosphere of 
community and improvement in such a high-stakes environment. 

In our department, both the responsive and connoisseurship models had 
been combined with the decision-making assessment model that links the results 
of the assessment process with departmental decision making about academic 
program development. 
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A fourth model, the goal-based model, assures systematic attention to 
the perceived performance of a given program when compared to the original 
programmatic intention (Conrad & Wilson, 1986) . The goal-based model also is 
popular because it focuses on the learner by relating outcomes to program 
goals and individual student performance. As is common in academic settings, 
our department utilizes a variety of means to assess individual students; 
grading student work as courses are completed, applying examinations of 
various types, conducting comprehensive examinations both oral and written to 
provide the student with an opportunity to demonstrate knowledge acquired 
through the program overall, and supervised internship/ field experiences. 
However, the relationship of assessment strategies to programmatic goals was 
previously ill-defined. 

In addition to attempting to determine what model of outcomes assessment 
should be applied to the process we were developing, the faculty also examined 
how quality should be defined. We considered four main perspectives: 
reputational, resources, outcomes, and value-added. 

The reputational perspective on quality had been commonly implemented in 
the application of the connoisseurship model. Quality from the reputational 
perspective is achieved when an external expert affirms its existence. 
Although by tradition the reputational perspective of quality is highly 
regarded, skepticism is now emerging. Critics have begun to make accusations 
of "cronyism" within a discipline or professional field. 

The resources perspective on quality focuses on input analysis. The 
physical, human, and financial assets are examined by internal or external 
reviewers. If, in the reviewers* judgment, the students are excellent 
demonstrated by rigid admission criteria, the facilities and equipment are 
modern and in working order, and the faculty are well-paid, have appropriate 
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credentials, and are well-published, quality has been achieved. A limitation 
of the resources perspective is that there is no determination as to whether 
or not the inputs are put to good use. Essentially the resources perspective 
of quality focusing on the presence of inputs avoids the issue of outcomes 
entirely. A limitation that has not escaped higher education critics who are 
increasingly demonstrating unwillingness to improve inputs through increased 
financial support to educational institutions at all levels. 

The outcomes perspective on quality focuses on the students who emerge 
from the educational experience. Quality within this view is most closely 
related to productivity analysis; the product being the student after 
graduation. Commonly, the outcomes perspective has been utilized in 
conjunction with the responsive model with the measure of quality being level 
of satisfaction among graduates and employers. Quality is achieved if high 
levels of satisfaction among graduates and employers is demonstrated. An 
obvious limitation of the outcomes perspective is in its operationalization. 
Highly satisfied graduates may not be highly able ones. A graduate that is 
highly satisfactory to an employer may be so for reasons that are utterly 
unrelated to prior academic experience. 

The value-added perspective of quality applies an industrial production 
model to student learning. In the value-added perspective of quality, student 
learning outcomes are re-conceptualized and an attempt is made to avoid the 
limitations of the sat isf action-equals-quality operationalization. Value- 
added tied to student learning outcomes is the difference between a student* s 
knowledge and skills at admission and their knowledge and skills at the time 
of graduation. It is a measure of the increase in student knowledge that has 
occurred during the academic program. While the value-added perspective of 
quality enjoys present popularity, the pre-test and post-test design that is 
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implicit in the definition is difficult to apply with rigor in university 
settings . 

Outcomes Assessment and Faculty Evaluation and Reward Mechanisms 

The information we had obtained from our university suggested to us that 
a goal-based model of outcomes assessment combined with a value-added 
perspective on the definition of quality was intended. We were encouraged to 
review our mission, goals, and objectives and to relate these programmatic 
facets to individual student learning outcomes. The process had been 
identified to us as essentially formative in nature; the outcome assessment 
effort was to be designed to assist us in achieving continuous internal self- 
improvement and was to be divorced from the processes of individual faculty 
evaluation, promotion, and tenure. 

While the outcomes assessment process we are developing is centering on 
internal self-improvement, it can not be denied that much of the pressure for 
improvement at our institution and in higher education generally has been 
externally generated and is often targeted at faculty. "Much of the recent 
condemnation of higher education centers around the belief that professors 
receive high pay for little work." (Murray, 1995, p.82) and "Parents are not 
at all sure that the effectiveness of instruction warrants the high costs of 
college education." (Centra, 1979, p.2) "Today, faculty members are asked 
more and more frequently to demonstrate the ways in which they are investing 
themselves in their institutions and professions and thus the ways in which 
they are fulfilling society’s investment in them." (Braskamp and Ory, 1994, p. 
xiii) In this environment, it is feasible that the effort to assess student 
learning outcomes will blend into the pool of other academic accountability 
requirements. Although our university is striving to separate the outcomes 
assessment process and the evaluation process. Centra (1979) suggests that 
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this distinction is neither necessary nor desirable, "Faculty members are 
evaluated in order to decide whether they should be promoted or rewarded and 
to improve their performance - two purposes that need not be mutually 
exclusive . " (p . 1) 

Envisioning a process of internal self-improvement of our program that 
is tied to student learning outcomes necessitates instructional improvement. 
It is difficult for us to envision improved instruction without imagining that 
more attention to the person who is teaching will also result. "The teacher 
or professor has, for hundreds of years, been the center of the instructional 
process,..." (Bryant, 1994, p.20) and as a faculty we are committed to 

improved instruction. 

However, as individuals we are also required to achieve a balance among 
teaching, research and service activities within evaluation, promotion, tenure 
and reward mechanisms that have become increasingly summative in nature over 
time and that are not evenly balanced among the varied responsibilities of the 
faculty. In addition, institutions are limited in their ability to provide 
monetary incentives for good teaching since faculty monetary rewards compete 
with other priorities for limited resources . 

The effectiveness of monetary rewards to enhance faculty performance, 

even if resources were available, can also be limited by faculty perceptions 

about the evaluation system. Braskamp and Ory (1994) used the results of a 

1989 survey by the Carnegie Foundation for Advancement of Teaching, to show 

that a majority of faculty members are uncomfortable with current methods of 

performance evaluation, (p . 5) These authors state, "It's administratively 

unwise to ignore the fact that a significant number of faculty are 

* 

dissatisfied with the current system." (p.7) 

While initially institutions focused on formative or developmental 
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features of faculty evaluation, legal decisions that have created constraints 
on this purpose and the desire to achieve consistency in personnel decisions 
have both resulted in the process becoming increasingly summative in nature. 
(Me Gee, 1995) As universities use the faculty evaluations process more for 
summative purposes than formative purposes, faculty may focus on workload 
functions for which they receive the greatest rewards to enhance their 
likelihood of achieving promotion and tenure. In most institutions this is 
for independently performed research. As a result, it is difficult to achieve 
the increased emphasis on instructional functions necessitated by the self- 
improvement orientation of outcomes assessment when faculty continue to be 
pressured to produce research funding and publications to survive within the 
overall faculty evaluation and promotion processes. The unanswered questions 
remains - how can faculty be encouraged to embrace the process and value 
student learning outcomes assessment when their promotion and tenure depend 
more heavily on their research activities than on teaching? 

The criterion-based faculty evaluation form used in the department of 
Educational Management and Development at New Mexico State University awards 3 
points for each three credit hour course taught, while faculty receive 9 
points for a single independently produced article in a refereed journal. 
Faculty futures are more dramatically influenced on the evaluation by 
concentrating effort on solitary research activity than devoting time to 
cooperative efforts to improve instruction. 

New Mexico State University, a Carnegie I research university, is not 
unique in its environment of "publish or perish" and the university does offer 
recognition and monetary awards to support faculty teaching efforts as do 
other high-quality oriented universities. The University of California at 
Berkeley, widely recognized for it academic excellence was recently ranked 4 
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out of 229 universities on academic reputation (U.S. News and World Report , 
September 18, 1995) and is considered proactive in their efforts to award 
quality instruction. 

In a recent article, Swartz, (1992) interviewed award winners from the 
past decade of the Distinguished Teaching Award at UC-Berkeley to determine 
whether there was a correlation between the award and faculty advancement, 
such as promotion or merit increase in salary. Swartz focused on the 
perception of the awardees and he found that teaching awards had very little 
or no positive affect on faculty advancement. From his observations, Swartz 
suggests that faculty may well conclude that it is in their own best interests 
- at least financially - to devote their energies more to research than to 
teaching . 

Swartz (1992) recorded the comments made by the Distinguished Teaching 
Award at UC-Berkley and they are revealing about the realities of devoting 
effort to instruction within the current faculty evaluation, promotion, and 
tenure processes. One recipient commented, "I would imagine that the award 
has only influenced advancement or merit increase in a minor way .... My own 
take is that if someone is a bad teacher, it will hurt them more than being a 
good teacher will help them." 

Another recipient remarked on the pressure to produce and publish 
research results, "I find an almost negative correlation. Research is 
rewarded, not teaching, and one is well advised not to become known as a 
brilliant* teacher unless one already has tenure. This sounds cynical, but I 
think it is realistic." (Swartz, 1992, p. 34) Another recipient remarked on 
efforts to solicit support for self improvement, "After receiving the award 
and starting a quite successful term as chair, I requested: (1) an 
acceleration; (2) a computer; (3) supplementary support for my sabbatical. 
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All three were refused. When I complained about the last, the dean informed 
me that teaching and service do not count at all on such decisions." 

Swartz* s (1992) findings demonstrate a reality of faculty work-life that 
can in the long run influence the good intentions inherent in outcomes 
assessment defined as continuous self improvement tied to student learning 
outcomes. The difficult question remains - how can faculty be encouraged to 
accept the self improvement purpose of student learning outcomes assessment 
when the faculty evaluation process rewards research more than teaching? As 
faculty continue to strike a balance in teaching, research and service - will 
the reward systems at higher education institutions be transformed to ensure 
that faculty can devote the time and effort required to consistently improve 
quality in classroom instruction? 

Transforming Outcomes Assessment Into Meaningful Practice 

As we examined the varied outcomes assessment models that been 
previously used by our department, the requirements of external accrediting 
agencies and professional organizations, the various perspectives on how 
quality should be defined and measured, and the most recent institutional 
requirements for outcomes assessment that is goal-based, student learning 
focused, and. centered on continuous self improvement, we realized that we were 
confronting "a new animal". To an extent our prior reliance on the 
connoisseurship and responsive models of assessment constrained our thinking. 
At times in our discussion we found ourselves trying to beat square pegs into 
round holes . We would fall back to familiar models that could not further our 
efforts to tie goals to student learner outcomes with existing assessment 
strategies. While frustrated by trying to fit square pegs into round holes, 
the process has helped us to identify areas for future improvement in our 
program construction, course content, field experience and supervision, and 
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assessment techniques. While resolution of these matters can not all be 
accomplished in the short-term, the identification of areas for self- 
improvement and the prioritization of items for immediate and future action 
suggest that one of the promises of outcomes assessment, continuous self 
improvement, will be realized over time. 

We also recognized that there were familiar and complementary tools 
available to us as we attempted to address the task of transforming a 
different outcomes assessment model into meaningful practice. While we could 
no longer rely on the connoisseurship and responsive assessment models we had 
previously applied, prior work related to departmental mission, goals, and 
objectives facilitated our effort to tie existing student assessment practices 
more directly to programmatic goals. In addition, our departmental efforts 
were facilitated by our own competence and expertise in program planning, 
analysis, and evaluation. To an extent, this new challenge provides an 
opportunity for us to practice what we preach in our professional and academic 
practice as educational administrators, consultants, and teaching faculty. 

To evaluate student learner outcomes related to programmatic goals, we 
are conducting a review of syllabi, assigned individual student and group 
activities, and assessments for each of our courses. Recognizing the 
continuing influence of accrediting agencies, legal entities, and professional 
organizations, we are also charting departmental goals, student learner 
outcomes, and assessments in relation to applicable external standards. While 
this task is time-consuming, it has facilitated our efforts to identify 
desirable additions to course content, unnecessary content duplication among 
courses, desirable alternative assessment strategies and ways in which more 
consistent instructional strategies and academic standards among our courses 
and varied faculty can be achieved. 
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There continue to be unresolved tensions and conflicts as we pursue our 
work. It is a struggle to retain student learning outcomes as a central 
concept not to be confused with traditional institutional analysis of faculty 
productivity and program effectiveness. 

While the promise of continuous self improvement is emerging, it is not 
yet clear that the promise will be fulfilled. To a large extent routine, 
bureaucratic challenges dominate our process. We are actively engaged in 
producing documents that we believe are desired by the institutional outcomes 
assessment process. While we are actively engaged in identifying areas for 
improvement, whether we as a faculty can realize outcomes assessment goals 
ultimately improving the number of students meeting the student learning 
outcomes we have identified through improved classroom and advising strategies 
is uncertain at this time. 

It is not yet clear how the institution will utilize the outcomes 
assessment process and the resulting information. While the institution has 
identified internal self-improvement goals, the possibility that additional 
levels or types of information analysis might result in future negative 
consequences for individuals and/or departmental programs creates ambiguity 
about future commitment. If information gathered to foster formative purposes 
is put to summative use, defensive reaction could circumvent the self 
improvement process . 

It is not yet clear how the department will utilize the information 
realized from the process. As a faculty, we will have to identify for 
ourselves how the evidence of student learning outcomes will be connected to 
our own and each others instructional strategies. The uneasy and perhaps 
unresolvable tension between formative and summative assessment applies to the 
department level just as it does to the overall institution. 
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As an academic department responding to the challenges of a new vision 
of outcomes assessment, we are attempting to respond to 
institutional/centralized demands for what is ultimately a decentralized 
process of improvement. In the reality of academic life, self improvement is 
not only a decentralized process - it is an intensely personal process. As 
individuals within our department, each of us must achieve a balance among the 
service, research, and teaching functions of our position. Despite our 
individual and collective commitment to quality instruction, the tension 
between an outcomes assessment process that is intensely focused on 
instruction and faculty evaluation, promotion, and tenures processes that are 
focused on independent research may also circumvent future improvement. 

While our progress in responding to the new challenges of outcomes 
assessment is very promising, our ability to predict the future is severely 
limited. It is uncertain that the tensions and conflicts between outcomes 
assessment and faculty evaluation, promotion, and tenure processes will, or 
even can, be resolved to assure that the promise of continuous self 
improvement will be fulfilled. Despite this uncertainty, our outlook remains 
positive. Self improvement in our departmental program and instructional 
activities will occur. The unanswered question concerns the magnitude of 
improvement we can achieve and for how long. 
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